Multi-Modal Learning for Dynamic Tactile Sensing

Dynamic tactile sensing allows humans to infer surface and material properties from the vibrations caused by the sliding motion between the skin and an object [1]. For example, one can easily determine the roughness of a surface by sliding one’s finger tip over the surface [2]. This sensory modality is also fundamental for tool usage, as it detects the vibrations resulting from the tool making contact with another object [3], and senses incipient slip between the fingers and the tool in order to maintain a good grip [4]. Robots performing dexterous manipulation tasks could therefore gain similar benefits from using dynamic tactile sensing [5], [6], [7]. However, raw time-series data received from dynamic tactile sensors is usually noisy and high-dimensional. The signal will also often contain confounding vibrations from other sources, e.g., the robot’s own vibrations [6]. Hence, it is difficult to directly use this sensory information to discriminate between different surfaces. Instead, we propose first learning a lowdimensional representation of the data. This low-dimensional space should capture the vibrations caused by the textured surfaces, but exclude dimensions containing the additional vibrations that are irrelevant to the tactile sensing task. In order to determine which components of the signal correspond to the textured surface, we employ a humaninspired approach. Humans are capable of combining information from multiple sensor modalities, such as touch, vision, and audition [8], [3]. Similarly, robots can learn a suitable low-dimensional representation for dynamic tactile data, by using vision information. In particular, the robot can determine relevant features by finding correlations between the vision and the tactile data of various textured surfaces. Parts of the data relating to the texture of the surface will occur in both sensor modalities. However, the additional vibrations detected by the dynamic tactile sensor will not be captured by the images of the objects, and hence these components will be excluded. In this work, we present two algorithms for learning lowdimensional representations of dynamic tactile data. These methods use the correlations between tactile and vision data taken from the same surfaces in order to determine which dimensions are relevant to the tactile sensing task. Once a lower-dimensional space has been learned, the robot can directly represent new tactile data in this space without requiring corresponding vision data. Using the proposed approach, the robot (see Fig. 1) was able to accurately discriminate between various textured surfaces using only the basic dynamic tactile sensor shown in Fig. 2. Figure 1. Robot performing tactile exploration of a textured surface.